MDL Procedures with `1 Penalty and their Statistical Risk

We review recently developed theory for the Minimum Description Length principle, penalized likelihood and its statistical risk. An information theoretic condition on a penalty pen(f) yields the conclusion that the optimizer of the penalized log likelihood criterion log 1/likelihood(f) + pen(f) has risk not more than the index of resolvability, corresponding to the accuracy of the optimizer of the expected value of the criterion. For the linear span of a dictionary of candidate terms, we develop the validity of description-length penalties based on the `1 norm of the coefficients. New results are presented for the regression case. Other examples involve log-density estimation and Gaussian graphical statistical models.